The approaches described in this section could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
Computer software development often involves several progressive phases such as definition, coding, quality assurance (QA) and testing, error removal or “debugging,” and maintenance. In commercial software development, testing and debugging often consumes considerable time, and may account for the largest time component in a development project. Traditional bug fixing requires detailed communication between testers and developers. Time is often wasted going back and forth between testers and developers trying to reproduce and isolate problems. Verifying that a bug has been fixed is error-prone and time consuming. Therefore, software developers are continually looking for ways to reduce the amount of time spent on testing and debugging.
One approach is to outsource QA and testing to test personnel in another location, even another country. However, outsourcing can involve language barriers and communication barriers when a developer prepares and sends written bug reports in a local language to QA personnel in another country who speak another language. Distance barriers, and complications arising from having developers and testers in different physical locations or even different time zones also can exist. Developers would like to have a software testing solution that facilitates communication in an outsourced environment.
Scheduling QA testing within the software development cycle can be difficult. Finding and fixing bugs is unpredictable, at best. Delays in QA testing can lead to late releases, missed market windows, and lost revenue. These issues may be acute in the fast-paced environment involved in developing computer games for platforms such as PC, Microsoft XBOX family, Sony Playstation family, Nintendo, etc.
To address these issues, several types of program execution recording systems have been developed. Generally, program-recording systems record information about a program under test as the program executes, and provide reports about that information. However, consistently reproducing bugs is a serious problem in software development, and achieving it typically involves expending significant resources. Some systems facilitate replaying program execution on a repeated basis. Using these systems, debugging is improved because bugs are reproducible by replaying a particular program execution session.
Types of program recording systems include external I/O recorders, source code instrumenters, and binary patching systems. Generally, external I/O recorders create and store recordings of network I/O operations, user input, and graphics or display 3-D (D3D) information. External I/O recorders may be use to simulate such external input to a program. External I/O recorders do not require any modifications to program code, are robust in response to code and asset changes, and can be used for cross-platform testing and demos. However, external I/O recorders may not produce accurate program replays due to timing issues, such as irregular network delays, or race conditions. Further, external I/O recorders may be unusable with some platforms for security reasons. External I/O recorders do not account for non-determinism in programs associated with thread context switching. External I/O recorders are sometimes termed macro recorders. A commercial example is Mercury Interactive's WinRunner.
Source code instrumenters usually provide proxy API libraries and modules; a developer must include the libraries or modules in source code for testing purposes. Source code instrumenters are somewhat reusable, can be easily expanded and tuned, and recordings may be portable across platforms. However, source code instrumenters are applicable only to modules for which program source code is available; they cannot be used to debug programs for which only executable machine code is available. Source code instrumenters typically require the use of a specialized API for certain calls by the developer, or a code-parsing module. Thus, the developer shoulders the burden of inserting the correct API calls in the source code of the program under test. Further, source code instrumenters provide no support for third-party modules such as dynamic linked libraries (DLLs) or linked executables, because such modules will not contain the required API calls at the time of testing. Source code instrumenters may not provide 100% accurate replays due to the effect of external events that are not trapped and recorded.
Binary patching systems operate by adding specialized recording code to the binary machine code of a program under test after compilation. A commercial example is Rational Purify. Binary patching systems are highly reusable and can produce accurate recordings by capturing detailed operational data. Binary patching systems do not require source code modifications, and can be applied to any executable, library or DLL. When disabled, binary patching systems do not affect program execution or size. Binary patching systems can capture low-level program calls, e.g., calls to hardware elements.
However, binary patching systems can be fragile when code or assets change. A recording of a program of a first version may be incompatible for replay when the program is modified to a later version. Binary patching systems may require special support for certain APIs, such as those relating to networking. Binary patching systems typically require special support for different processors having different machine instruction sets, and for different binary file formats (e.g., PE, XBE, ELF). Binary patching systems do not readily produce recordings that are portable across platforms. Further, cross-module inlining of code (e.g., using Link Time Code Generation (LTCG)) can distort function boundaries and make patching inaccurate.
In addition, known binary patching systems are not capable of recording all sources of non-determinism that may exist in an application.
Based on the foregoing, there is a clear need for an improved way to identify and reproduce bugs in a computer program that is undergoing development, QA or test.
Users in this field would appreciate having a solution that could save money in the process of bringing a product to market and shorten the software development cycle. Users also seek a solution that has little impact on existing software development workflows. For example, users would prefer a solution that does not require developers to use new application programming interfaces (APIs) and that does not impose new requirements on the development process. Users also wish to have solutions that facilitate outsourcing by eliminating the need for detailed bug reports, add predictability to scheduling QA testing, and optimize the QA process.